Visual object tracking has actively been researched for several decades. Depending on the prior information about the target category, the tracking algorithms are usually classified as category-free methods and category-based methods. However, the prior works were developed using conventional handcrafted features, which have difficulties of being scaled up.